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- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 
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a plurality of predetermined semantic labels". This means that one region (of the RAG) 
is associated with only one semantic label, Modestino et a! teachings clearly disclose 
this concept (Figures 1(a)-1(c), 5(a)-5(d) and 6(a)-6(d); Tables l-IV; Page 606. Section 
I, Introduction, Paragraphs 1-3; Pages 610-613, Sections V-VII, Modestino et al 
disclose a Markov Random Field (MRF) model-based approach image interpretation 
and classification by performing knowledge-based high-level processing via assigning 
higher-level expressions (Stereotypes) to the classified objects. The region adjacency 
graph is classified based on higher-level expression or stereotypes (e.g., rural road 
scenes comprising road, field, car and sky). "Two regions" and "three regions" 
categories depicted in Table I in association of Figures 1(a)-1(c) are stereotype or 
higher-level expressions consistent with the definition of "Stereotype" in the instant 
application Page 7, Lines 11-17. Modestino et al, furthermore, clearly contrast low-level 
processing based on the image primitives such as edges and regions, and high-level 
processing based on higher-level expressions (Page 606, Section I, second Paragraph). 

Claim Rejections - 35 USC § 102 

4. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that 
form the basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(b) the Invention was patented or described in a printed publication in this or a foreign country or in public 
use or on sale in this country, more than one year prior to the date of application for patent in the United 
states. 

5. Claims 1, 3-10, 18, 20-27, 35, 37-44, 52. 54-61. 66. 68-75 and 80, 82-89 are 
rejected under 35 U.S.C. 102(b) as being anticipated by Modestino et al (IEEE Paper 
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ISBN: 0162-8828; A Markov random Field Model-Based Approach to Image 



Interpretation). 



Regarding Claim 1, Modestino et a! disclose a method of classifying a digital 



image, said method comprising the steps of: 

segmenting the digital image into substantially homogeneous regions (Figure 1; 
Pages 607-608, Section ll-B. The image in Figure 1a is segmented into homogeneous 
nodes or regions, R = {Ri, R2, Rn) as depicted in Figure 1(c).); 

processing the regions to provide a region adjacency graph for the digital image, 
the region adjacency graph representing adjacencies between the regions wherein at 
least one of the regions of the region adjacency graph is associated with at least one of 
a plurality of predetermined semantic labels (Figure 1 ; Page 607, Column 2, Section IIB, 
first Paragraph. G = {R, £} is the region adjacency graph comprising of the set of nodes 
R = {Ri, R2, Rn} and the set of edges E connecting the regions. The labels assigned 
to the segmented regions ); 

analyzing the region adjacency graph to identify predetermined patterns of the 
semantic labels associated with the regions (Figure 1; Page 607, Column 2, Section IIB, 
second Paragraph. L = {Li, L2, Lm} is the set of all predetermined patterns of 
regions. Abstract, Lines 3-10; Figure 1; Pages 607-608, Section ll-B. The image in 
Figure la is segmented into homogeneous nodes or regions, R = {R7, R2, Rn) as 
depicted in Figure 1(c). Predetermined patterns of the semantic labels are identified as 
shown in Tables I through IV.); and 

assigning one of a plurality of predetermined stereotypes to the digital image 
according to each identified pattern of the semantic labels associated with the regions. 
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such that the assigned stereotype represents a classification of the digital image 
(Figures 1, 5(a)-5(d), 6(a)-6(c); Tables l-IV. The region adjacency graph is classified 
based on higher-level expression or stereotypes (e.g., rural road scenes comprising 
road, field, car and sky. "Two regions" and "three regions" categories depicted in Table 
I in association of Figures 1(a)-1(c) are stereotype or higher-level expressions. The 
digital image is classified based on the identified patterns (features) of the regions as 
depicted in Figures 1, 5(a)-5(d), 6(a)-6(c) and Tables l-IV,). 

Regarding Claim 3, Modestino et al further disclose the method according to 
Claim 1 , wherein the digital image is classified on the basis of a size of one or more 
regions of the digital image (Tables l-IV. Classification is based on the areas and the 
boundary lengths of the regions identified by the RAG nodes.). 

Regarding Claim 4, Modestino et al further disclose the method according to 
Claim 3, wherein the digital image is classified on the basis of an adjacency of the 
regions (Figures 1 and 6(b). Region adjacency graphs are inherently generated based 
on the spatial dependence of objects in the relative proximity of each other as depicted 
in Figure 1(c).). 

Regarding Claim 5, Modestino et al further disclose the method according to 
Claim 1 , wherein the digital image is classified on the basis of semantic label content of 
the region adjacency graph (Page 613, Table lll(b). Stereotypes are assigned to region 
adjacency graph based on the semantic label content, i.e., "road, field" or "sky, field" as 
depicted in Table lll(b).). 

Regarding Claim 6, Modestino et al further disclose the method according to 
Claim 1 , wherein the digital image is classified on the basis of a mean color of one or 
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more regions of the digital image (Page 613, Table 1 11(a), Average Gray Level of the 
regions). 

Regarding Claim 7, Modestino et al further disclose the method according to 
Claim 1 , wherein the plurality of stereotypes are stored in an association lookup table 
(Page 607, Column 2, Section MB, second Paragraph. Set of all interpretation labels L 
is the database or lookup table for stereotypes or node labels.). 

Regarding Claim 8, Modestino et al further disclose the method according to 
Claim 1, wherein the stereotypes are represented in a hierarchal arrangement (Figure 1; 
Tables 1(a) and 19b). Figure 1(c) represents the nodes of stereotype in a hierarchical 
arrangement based on the number of neighboring regions. Based on associated 
cliques. Sky, Road and Car are represented in a hierarchical arrangement.). 

Regarding Claim 9, Modestino et al further disclose the method according to 
Claim 7, wherein each of the stereotypes has a hierarchical path arrangement (Figure 1 . 
Sky, Road and Car has a hierarchical path arrangement. Same concept is applicable to 
the higher-level expressions or node labels in Tables lll(b) and IV(b).). 

Regarding Claim 10, Modestino et al disclose the method according to claim 1, 
wherein the region adjacency graph is provided by analyzing contextual data associated 
with one or more regions of the digital image (Figure 1, Page 607, Section MB, first 
Paragraph). 

With regards to Claims 18, 35, 52, 66 and 80, arguments analogous to those 
presented for Claim 1 are applicable to Claims 18, 35, 52, 66 and 80, Concerning 
Claims 52, 66 and 80, Modestino et al further disclose providing a set of labeled regions 
(Abstract Lines 5-10; Page 607, Column 2, Section IIB, second paragraph, set of labels 
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L = {Li, L2, Lm}). Markov Random Field (MRF) model-based approach segments the 
image into a collection of disjoint regions that form the nodes of an adjacency graph. 
Once the adjacency graph has been determined, image classification will be achieved 
through assigning object labels to the segmented regions using domain knowledge, 
extracted feature measurements, and spatial relationship between the various regions 
(Abstract, Lines 5-10). 

With regards to Claims 20, 37, 56, 70 and 84, arguments analogous to those 
presented for Claim 3 are applicable to Claims 20, 37, 56, 70 and 84. 

With regards to Claims 21 , 38, 55, 69 and 83, arguments analogous to those 
presented for Claim 4 are applicable to Claims 21, 38, 55, 69 and 83. 

With regards to Claims 22, 39, 54, 68 and 82, arguments analogous to those 
presented for Claim 5 are applicable to Claims 22, 39, 54, 68 and 82. 

With regards to Claims 23, 40, 57, 58, 71, 72, 85 and 86, arguments analogous 
to those presented for Claim 6 are applicable to Claims 23, 40, 57, 58, 71 , 72, 85 and 
86. 

With regards to Claims 24, 41, 59, 73 and 87, arguments analogous to those 
presented for Claim 7 are applicable to Claims 24, 41 , 59, 73 and 87. 

With regards to Claims 25, 42, 60, 74 and 88, arguments analogous to those 
presented for Claim 8 are applicable to Claims 25, 42, 60, 74 and 88. 

With regards to Claims 26, 43, 61 , 75 and 89, arguments analogous to those 
presented for Claim 9 are applicable to Claims 26, 43, 61 , 75 and 89. 

With regards to Claims 27 and 44, arguments analogous to those presented for 
Claim 10 are applicable to Claims 27 and 44. 
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Claim Rejections - 35 USC § 103 



6. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

7. Claims 1 1 , 17, 28, 34, 45, 62-65, 76-79 and 90-93 are rejected under 35 
U.S.C. 103(a) as being unpatentable over Modestino et al in view of Li et al (U.S. 
5,930,783). 

Regarding Claim 1 1 , Modestino et al do not explicitly disclose the method 
according to Claim 10, wherein the contextual data comprises information generated by 
one or more separate sources of the information. 

Li et al disclose a semantic and cognition based image retrieval methodology 
comprising analyzing contextual data generated by one or more separate sources of 
information (Figure IB, Semantic-based Query, Cognition-based Query; Column 12, 
Lines 30-40). 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify Modestino et al invention according to the teachings of Li 
et al to analyze contextual data generated by one or more separate sources of 
information because it will expand versatility of the image segmentation and 
classification. It will utilize the advantages of image retrieval based on both image 
semantics and visual examples of the image (Li et al, Column 3, Lines 27-35). 
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Regarding Claim 17, Modestino et al do not explicitly disclose the method 
according to Claim 1 , wherein said digital image is stored in a database of digital 
images and wherein said classification can be used to retrieve said digital image from 
said database. 

Li et al disclose a semantic and cognition based image retrieval methodology 
wherein the digital image is stored in a database of digital images and wherein the 
classification can be used to retrieve said digital image from the database (Column 4, 
Lines 32-50). 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify Modestino et al invention according to the teachings of Li 
et al to use classification to retrieve the digital images stored in a database because it 
will utilize a plurality of information sources for image retrieval including semantic- 
based, cognition-based and scene-based queries thereby eliminating weakness of the 
individual approaches (Li et al; Column 3, Lines 27-35). 

With regards to Claims 28 and 45, arguments analogous to those presented for 
Claim 1 1 are applicable to Claims 28 and 45. 

With regards to Claim 34, arguments analogous to those presented for Claim 17 
are applicable to Claim 34. 

With regards to Claims 63, 77 and 91 , arguments analogous to those presented 
for Claim 17 are applicable to Claims 63, 77 and 91 . Li et al retrieve the digital image 
by using a keyword representing a stereotype (Column 12, Lines 30-50). 

With regards to Claims 64, 78 and 92, arguments analogous to those presented 
for Claim 17 are applicable to Claims 64, 78 and 92. Neither Modestino nor Li et al 
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explicitly retrieve the digital image by using an icon to represent a stereotype. Utilizing 
an icon to represent a keyword is extremely well known in the art (Official Notice). 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify Modestino et al and Li et a! combination to Utilize an icon 
for representing a keyword because icons are a significant factor in the user- 
friendliness of graphical user interface that serve as visual mnemonics to allow the user 
to control certain computer actions without having to remember commands or type them 
at the keyboard. 

With regards to Claims 62, 76 and 90, arguments analogous to those 
represented for Claim 64 concerning utilizing an icon is applicable to claims 62, 76 and 
90. 

With regards to Claims 65, 79 and 93, arguments analogous to those presented 
for Claim 17 are applicable to Claims 65, 79 and 93. Li et al retrieve the digital image 
by using a keyword representing a generalization of a stereotype (Column 12, Lines 30- 
50). 

8. Claims 12, 29 and 46 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Modestino et al further in view of Li et al (U.S. 5,930,783) and Jain et al (US 
6,360,234). 

Regarding Claim 12, neither Modestino et al nor Li et al disclose the method 
according to Claim 1 1 , wherein a corresponding portion of said contextual data is 
obtained from a temporal region of interest for each source of said information, 

Jain et al disclose a method for video cataloging by providing metadata 
associated with the image wherein a corresponding portion of contextual data is 



Application/Control Number: 09/395,993 Page 1 1 

Art Unit: 2623 

obtained from a temporal region of interest for each source of information (Figures 6-9; 
Column 6, Lines 30-67). 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify Modestino et al and Li et al combination according to the 
teachings of Jain et al to obtain a corresponding portion of the contextual data from a 
temporal region of interest for each source of information because it will accurately 
classify and intelligently extract information, termed metadata, about the contents of 
video stream in real time (Jain et al. Column 1 , Lines 46-49). 

With regards to Claims 29 and 46, arguments analogous to those presented for 
Claim 12 are applicable to Claims 29 and 46. 

9. Claims 13-16, 47-51 and 30-33 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Modestino et al in view of Jain et al (US 6,360,234), 

Regarding Claim 13, Modestino et al do not disclose the method according to 
Claim 1, further comprising the step of providing metadata associated with the digital 
image, wherein the metadata includes the stereotypes of the digital image. 

Jain et al disclose a method for video cataloging by providing metadata 
associated with the image comprising the step of providing metadata associated with 
the digital image, wherein the metadata includes the stereotypes of the digital image 
(Figures 16 and 17; Column 13, Lines 52-67, Column 14. Lines 1-31). 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify Modestino et al invention according to the teachings of 
Jain et al to provide metadata associated with the digital image, wherein the metadata 
includes the stereotypes of the digital image because it will improve image appearance 



Application/Control Number: 09/395,993 Page 12 

Art Unit: 2623 

and enhance image classification. It will accurately classify and intelligently extract 
information, termed metadata, about the contents of video stream in real time (Jain et 
al, Column 1, Lines 46-49) 

Regarding Claim 14, Jain et al further disclose the method according to Claim 13, 
wherein the metadata includes a hierarchical path associated with the respective 
stereotype of each digital image (Figures 9, 15-17; Table 1; Column 8, Lines 22-61). 

Regarding Claim 15, Jain et al further disclose the method according to Claim 14, 
wherein the hierarchical path is stored with a respective stereotype as a metadata 
object which is associated with a respective image object (Figure 6; Column 6, Lines 
29-38). 

Regarding Claim 16, Jain et al further disclose the method according to Claim 14, 
wherein the hierarchical path is stored as a referenced lookup table (Figure 7). 

With regards to Claim 47, arguments analogous to those presented for Claim 13 
are applicable to Claim 47. 

With regards to Claim 48, arguments analogous to those presented for Claim 14 
are applicable to Claim 48. 

With regards to Claim 49, arguments analogous to those presented for Claim 15 
are applicable to Claim 49. 

With regards to Claim 50, arguments analogous to those presented for Claim 16 
are applicable to Claim 50. 

With regards to Claim 51 , arguments analogous to those presented for Claim 17 
are applicable to Claim 51 . 
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Regarding Clainn 30, Modestino et al do not disclose the method according to 
Claim 18 further comprising the step of providing metadata associated with the digital 
image, wherein the metadata includes the stereotypes of the digital image. 

Jain et al disclose a method for video cataloging by providing metadata 
associated with the digital image, wherein the metadata includes stereotypes of the 
digital image (Figures 16 and 17; Column 13, Lines 52-67, Column 14, Lines 1-31). 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify Modestino et al invention according to the teachings of 
Jain et al to provide metadata associated with the digital image, wherein the metadata 
includes stereotypes of the digital image because it will accurately classify and 
intelligently extract information, termed metadata, about the contents of video stream in 
real time (Jain et al, Column 1, Lines 46-49). 

With regards to Claim 31, arguments analogous to those presented for Claim 14 
are applicable to Claim 31. 

With regards to Claim 32, arguments analogous to those presented for Claim 15 
are applicable to Claim 32. 

With regards to Claim 33, arguments analogous to those presented for Claim 16 
are applicable to Claim 33. 

Contact Information 

1 0. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Mehrdad Dastouri whose telephone number is (703) 
305-2438. The examiner can normally be reached on Monday to Friday from 8:00 a.m. 
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to 4:30 p.m. If attempts to reach the examiner by telephone are unsuccessful, the 
examiner's supervisor, Amelia Au can be reached on (703) 308-6604. 

The fax phone numbers for the organization where this application or proceeding 
is assigned are (703) 872-9314 for regular and for After Final communications. 

Any inquiry of a general nature or relating to the status of this application or 
proceeding should be directed to the receptionist whose telephone number is (703) 305- 
4700. 
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